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ABSTRACT 



This study was conducted to determine whether the writing 
skills, as measured by the University Writing Requirement (UWR) , of Kentucky 
high school graduates who had participated in Kentucky’s Commonwealth 
Accountability Testing System (CATS) writing portfolios had improved in 
comparison with their peers in the past who had not done the writing 
portfolios. The UWR is one of Eastern Kentucky University's standards for 
graduation. The objectives were to compare the UWR scores of pre- and 
post -Kentucky Education Reform Act (KERA) students and to determine the 
significance of the difference. Researchers randomly selected 50 UWR essays 
written by Kentucky high school graduates in response to a prompt, Spectator 
Sport, given in fall 1989 (pre-KERA) . The same number of essays responding to 
the same prompt given in summer 1997 (post-KERA) were also selected. A copy 
was made of each of the 100 essays to mask the effects of aging and to make 
the physical appearance of all the essays similar. Four UWR trained and 
experienced readers were asked to score the essays, and each essay was scored 
by two readers. The means of the essays in pre-KERA and post-KERA groups were 
computed and subjected to t- tests to determine the significance of the 
difference. No significant difference was found at any level between the 
means of the two groups. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



TM032142 



I 






00 



00 

^r 



Q 



W 



1 

Pre-and post KERA students’ writing skills... 



Pre-and Post KERA Students’ Writing Skills - A Comparative Study 



paper presented 



by 



Qaisar Sultana, Eastern Kentucky University 
Lisa Kay, Eastern Kentucky University 



$ 

'' 



U.S. DEPARTMENT OF EDUCATION 
Olfice of Educational Rosoarch and Improvomont 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

0r This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
j|t BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



The Twenty Ninth Annual conference of 
Midsouth Education Research Association 



Bowling Green, Ky. 
November 15-17, 2000 




2 



BEST COPY AVAILABLE 



2 

Pre-and post KERA students’ writing skills... 



ABSTRACT 

In 1989, Eastern Kentucky University (EKU) implemented the University Writing 
Requirement (UWR) as one of EKU’ s standards for graduation. The Kentucky Educational 
Reform Act (KERA) was passed in 1990. The center piece of KERA was educational 
accountability which is measured through a comprehensive assessment system now known as the 
Commonwealth Accountability Testing System (CATS). One component of this accountability 
system has been the writing portfolio. Since 1991 all fourth, eighth, and twelfth graders in 
Kentucky's public schools have been required to do the writing portfolio. 

This study was conducted to determine whether the writing skills, as measured by the UWR, 
of Kentucky high school graduates who underwent the CATS writing portfolios had improved as 
compared to their peers in the past who did not do the writing portfolios. Specifically, the 
objectives were to (a) compare the UWR scores of Pre-and post KERA students, and (b) 
determine the significance of the difference. 

The researchers randomly selected 50 UWR essays written by Kentucky's high school 
graduates on a prompt. Spectators Sport, given in Fall 1989 (Pre-KERA). The same number of 
essays on the same prompt given in Summer 1997 (post KERA) were also randomly selected. A 
copy of each of the 1 00 essays was made to mask the effects of aging and to make all the essays 
look exactly alike in their physical appearance. Four UWR trained and experienced readers were 
asked to score the essays. Each essay was scored by two readers. 

The means of the essays in the pre-KERA and post-KERA groups were computed and 
subjected to t tests to determine the significance of the difference. No significant difference was 
found at any level between the means of the two groups. 
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The publication of A Nation at Risk in 1983 created a nation-wide interest in improving 
education at all levels. This report contributed to a variety of efforts directed to reform education 
in the country. The Holmes Group and the Carnegie Foundation proposed different conceptual 
and theoretical models of education. Subsequently, some colleges and departments of education 
across the country underwent organizational and/or curricular changes. 

At Eastern Kentucky University (EKU), a small group of faculty members, concerned about 
the poor communication skills of the students, formed a committee. After a few years of 
deliberations, this group, called the University Writing Requirement Committee, proposed a 
University Writing Requirement (UWR) for all EKU students seeking a baccalaureate degree. 

The proposal was approved by the appropriate authorities of the university. It was implemented 
in 1 989. All undergraduate degree students at Eastern Kentucky University are now required to 
pass the University Writing Requirement (UWR). 

The UWR is administered each semester including summer. Students are required to take the 
UWR in the first semester following completion of 60 credit hours of course work. Transfer 
students who transfer 60 credit hours or more must take the exam in the first semester of 
enrollment. Students who fail the first attempt may retake the exam and continue taking courses 
in the following semesters) under certain conditions. (A complete copy of the UWR policy is 
contained in Tablel). 



Insert Table 1 about here 



The UWR consists of an essay written by the students on a prompt, which is created or 
selected by the UWR Committee. Occasionally a prompt is repeated. Students are given a four- 
page booklet with the prompt already printed on it. They are to identify themselves on the 
booklets only by their social security numbers. Students have an hour to write the essay. The 
essays are scored, according to a seven point rubric which is presented in Table 2, by a group of 
faculty members who have been trained in holistic scoring. Each essay is read by the first reader 
who records the score on the essay booklet. A secretary collects the booklets as they are read 
and blocks the scores. The booklets are then given to the second reader who records the scores 
on them. Each essay is scored by at least two readers and occasionally by a third reader when the 
scores given by the first two readers are not contiguous. Two readers’ scores are added and are 
recorded on the booklets as the final scores. When an essay is read by three readers the average 
of the scores is computed and multiplied by two. The total is recorded as the final score. 



Insert Table 2 about here 



The deliberations of UWR Committee and the implementation of UWR at EKU coincided 
with a major event in the Commonwealth of Kentucky. In November, 1985, a complaint was filed 
by 1 1 school districts in the Franklin Circuit Court in Kentucky challenging the equity and 
adequacy of funds provided to individual school districts by the Commonwealth. The “Circuit 
Court issued a judgment in October, 1988, stating that the General Assembly had failed to provide 
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an efficient system of common schools, and that the system of school financing was inefficient, in 
the constitutional sense, and discriminatory.” (KDE, 1994) 

“On appeal, the Kentucky Supreme Court issued an opinion in June, 1989, which held that the 
system of common schools in Kentucky was unconstitutional.” (KDE, 1994) In response to the 
ruling of the highest court in the Commonwealth, the General Assembly embarked on 
restructuring the entire public education system in Kentucky. It appointed a Task Force on 
Education Reform in July, 1989, composed of the leadership of the House and Senate and 
appointees of the governor. The recommendations of the Task Force resulted in House Bill 940, 
which was approved by the 1990 General Assembly. The Kentucky Education Reform Act 
(KERA) was signed by the governor on April 1 1, 1990. 

One of the key provisions of KERA is accountability. A testing program, Kentucky 
Instructional Retrieval Information System (KIRIS), made up of subject matter tests, performance 
events, and portfolios (including the writing portfolio) were administered to students in the fourth, 
eight, and twelfth grades during 1991-1992 school year to determine baseline data. “Portfolios 
occupied a key place in KIRIS, both as a means of assessment that directly tapped student work 
in classrooms, schools, and districts. Since the contents of the portfolios arose from student’s 
classroom work, the portfolio was the assessment component that most clearly reflected local 
curriculum and instruction.” (KDE, 2000) The statewide testing program, now known as the 
Commonwealth Accountability Testing System (CATS), continues to be administered each year 
and is used for rewards and sanctions awarded to schools across the commonwealth. The Writing 
Portfolio is now assessed in grades 4, 7, and 12. 

The Writing Portfolio, designed by a committee of Kentucky English/Language Arts 
educators, consists of a collection of students’ written products in broad categories: 

• Personal experience writing; 

• Imaginative writing; 

• Reflective writing; 

• Trans-active writing for real-world purposes and audiences. 

The Writing Portfolios are scored locally by school teachers who have been provided extensive 
training in portfolio scoring. Six criteria are applied holistically to produce a single final judgment 
which is Novice, Apprentice, Proficient, or Distinguish. The six criteria are: 

• Purpose/ Audience Awareness 

• Idea development/Support; 

• Organization; 

• Sentence Structure and Variety; 

• Language (Word Choice and Usage); and, 

• Correctness (spelling, punctuation, and capitalization). 

The statewide assessment system in Kentucky is designed to measure the schools’ success in 
the attainment of Kentucky’s Academic Expectations. One of the Academic Expectations is that 
all students should write for multiple purposes in multiple forms for a variety of audiences. 

The Writing Portfolio is calculated to assess this Expectation. 
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Purpose of the Study 

This study was conducted to compare the writing skills, as measured by UWR, of Kentucky 
high school graduates who had gone through CATS writing portfolio (post-KERA) with their 
peers prior to the implementation of KERA. Specifically, the objectives were to (a) compare the 
UWR scores of pre-and post KERA students, and (b) determine the significance of the 
difference. The null hypothesis was that there was no difference between the mean UWR scores 
of the two groups of students. 

Methodology 

The researchers reviewed the UWR records to find a prompt which was used prior to the 
enactment of KERA and which was repeated a few years after KERA. They found that Spectator 
Sports (copy of the complete prompt is presented in Table 3) was used in Fall 1989 and again in 
Spring of 1997. It was believed that the 1997 group of students included Kentucky high school 
graduates who had completed the KERA writing portfolio in the twelfth grade. The researchers 
went to the university archives and pulled out the UWR booklets, essays written by students in 
Fall 1989 and Spring 1997. The 1989 group was designated as pre-KERA and the 1997 group 
was designated as post-KERA. 



Insert Table 3 about here 



The researchers identified the social security numbers of Kentucky students and separated their 
essay booklets from the total number of booklets for both years, 1989 and 1997. Of these, 50 
booklets were randomly selected from each group. The social security numbers were recorded on 
a data sheet for each of the two groups separately. The scores recorded on each essay booklet 
and the dates were blocked. The booklets of the two groups were mixed together and a photo- 
copy of each booklet was made in order to eliminate the effect of aging on the booklets and to 
make them appear exactly alike. 

A research grant was submitted to the University for paying the readers. Pending approval of 
the grant, the researchers decided on the date of the scoring session and invited all the UWR 
readers to volunteer. They were informed that this reading was for a research study and that they 
may get paid contingent on the funding of the grant by the university. Six of them did. A scoring 
session was held on May 6, 2000, and six faculty members (excluding the researchers) read and 
scored the essays. Each essay was read by two readers, as is routinely done at UWR. No essay 
had to be read by a third reader. The score assigned to an essay by the first reader was blocked, 
just as is done at all regular UWR readings, before it was given to the second reader. The readers 
were not told the details of the research or its design until after the reading had been completed. 

The researchers added the scores given to each essay and computed the average of the scores 
given by the two readers just as it is done at regular UWR. The essays of the 1989 and 1997 
groups were separated according to the social security numbers which had been recorded on the 
data sheet prior to mixing the two groups. 
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The scores for both years were analyzed using statistical software. Summary statistics 
forthe pre-KERA and post-KERA scores are given in Table 4. 



Insert Table 4 about here 



A two-sample t-test to compare the means for 1989 and 1997 was performed. The results are 
summarized in Table 5. No significant difference between the means was found; the p - value of 

0.48 is very large. 



Insert Table 5 about here 



A 95% confidence interval for is even more informative: (-0.40, 0.84). Since it 

can be said with 95% confidence that p^ - p^, is between -0.40 and 0.84, it is clear that the two 
population means are likely to be very close, possibly even equal to each other. 

Discussion 

Accountability is the center piece of Kentucky’s educational reform. According to KERA, 
Successful schools will receive monetary rewards from the State; unsuccessful schools will be 
required to develop plans for improvement. If a school’s student performance declines 
significantly, it may be declared a school in crisis. Students attending schools in crisis may 
transfer to more successful schools and distinguished educators will be assigned to assist schools 
needing to improve. 

These schools will develop an improvement plan and qualify for implementation funds from the 
Commonwealth School Improvement Fund.” (KRS 158.6455; 158.782; 158.805) 

The CATS assessment is taken extremely seriously by every school in Kentucky because of the 
high stakes attached to it. Teachers spend an inordinate amount of time assisting the students 
write and rewrite their portfolio entries. Students are stressed out because of the demands made 
of them to do their Writing Portfolios well. 

Given the amount of instructional time spent on the Writing Portfolios, the results of this 
study were surprising to the researchers. Some factors may account for these surprising results. 
Grade level students write their portfolio entries on topics of their own choice. They have no 
time limit to write these entries. They have access to the spell check, grammar check, thesaurus, 
etc. They write their portfolio entries in their own natural classroom or home environment. They 
receive unlimited help from their peers, parents, and teachers in writing their portfolio entries. On 
the other hand, the UWR is highly structured. Students are required to write an essay on a given 
prompt. They also have a one-hour time limit. It is possible that under these structured 
circumstances they are not able to organize their thoughts and demonstrate their ability to express 
them in writing adequately. 

One goal of KERA was to provide one personal computer per six pupils in every school in 
the state. Kentucky has not fully met this goal yet but it is leading the nation in the number of 
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personal computers available in classrooms. It is probable that Kentucky’s high school graduates 
have become totally dependent on technology with regards to their writing mechanics. They have 
not really learned to write. 

The UWR scoring guide is more specific than the criteria applied in portfolio scoring. The 
Writing Portfolio criteria serve as the standard. It is possible that this standard is not sharp 
enough to adequately teach writing skills to students. Hence, no difference is seen in the mean 
UWR scores of pre-and post KERA high school graduates of Kentucky. 

Since 1985, Kentucky has had a New Teacher Internship Program. After graduation, during 
the first year of teaching, all new teachers are observed by a team of three including a teacher 
educator. The senior author been serving as a teacher educator on the new teacher internship 
committees every year. As such she spends a lot of time in schools. She reads lesson plans 
submitted by the new teacher interns. She reads the portfolio required of every new teacher 
intern. Regularly, she sees new teachers who are not familiar with word usage. They are 
deficient in applying rules of grammar. She also teaches three graduate courses a semester. And, 
all of our graduate students are full time teachers. Every semester there are students who make 
one wonder how they ever passed English 101. Teachers who have not learned the writing skills 
themselves cannot teach these skills to their students. It is probable that Kentucky students are 
receiving inappropriate instruction. 

The Portfolio Scores of Kentucky students show a steady improvement. These portfolios are 
scored by their own teachers. The Kentucky Department of Education has provided training to 
the teachers in portfolio scoring. The department also conducts a limited portfolio audits. The 
UWR essays are not scored by students’ own teachers. This may have some bearing on the 
results of this study. 

Conclusions 

From the results of this study, it appears that there are no differences in the writing skills 
of pre- and post-KERA students, as measured by the UWR. It should be noted that the post- 
KERA students in the study had completed only the high school writing portfolio. It is assumed 
that students completing the writing portfolio at the elementary, middle, and high school levels 
may indeed show improvement in their writing skills. This study needs to be replicated in the 
future to test this hypothesis. 
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